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(54) Method for enhancement of reduced color set images 



(57) A method of improving the range of color imag- 
es which can be processed by multiple enhancement 
processing elements, and improving image quality of 
limited color set images. The inventive method deter- 
mines (107) the number of colors that form an image, 



and with that information, adds a control function to one 
or more enhancement processors (112,122,132), 
which, based on a number of colors, increases and/or 
decreases the effect of the enhancement function. In do- 
ing so, the range of color images which can be handled 
is increased. 
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Description 

The present invention is directed toward a method 
and apparatus for improving the appearance of a digital 
image having a pictorial scene, and more particularly, 
toward improving images that are low color quality rep- 
resentations of natural scene images. 

In the past, a typical application for copiers or scan- 
to-print image processing systems was to reproduce an 
input image as accurately as possible, i.e., render a 
copy. Thus, copies have been rendered as accurately 
as possible, flaws and all. However, as customers be- 
come more knowledgeable in their document reproduc- 
tion requirements, they recognize that an exact copy is 
often not what they want. Instead, they would rather ob- 
tain the best possible document output. Until recently, 
image quality from the output of a copier or a scan-to- 
print system was directly related to the input document 
quality. One very common set of input documents in- 
cludes photographs. Unfortunately, photography is an 
inexact science, particularly among amateurs, and orig- 
inal photographs are often poor. Alternately, technology, 
age or image degradation variations often result in pic- 
tures having an unsatisfactory and undesirable appear- 
ance What is desired then, is a copy giving the best 
possible picture, and not necessarily a copy of the orig- 
inal. 

Image enhancement, as used herein, refers only to 
processes which improve the output quality of the im- 
age, and not to internal operations, necessitated by par- 
ticular processor limitations. Thus, for example, chang- 
ing TRC for better reproduction of an image is an en- 
hancement operation. However, reducing the number of 
colors representing image to place an image into a par- 
ticular file format, such as .GIF files, or BMP files, is not 
an enhancement operation. 

Photography has long dealt with this issue. Analog 
filters and illumination variations can improve the ap- 
pearance of pictures in the analog photographic proc- 
ess. Thus, for example, yellow filters enhance the ap- 
pearance of white clouds against a blue sky in black and 
white images. Further, various electrophotographic de- 
vices, including digital copiers, can clean up and im- 
prove ifriages by adjustment of thresholdrfiU^ 
background suppression. Generally, these methods are 
manual methods which a user must select on an image 
by image basis. Unfortunately, the casual user is not 
skilled enough to perform these operations. The inability 
to perform image enhancement operations is exacer- 
bated when additionally dealing with color controls. 

Three possible choices are presented by the art in 
the area of image enhancement. In the first case, we 
can do nothing. Such a system is a stable system, in 
that it does no harm to an image. This is a common ap- 
proach taken to reproduction. However, the output doc- 
uments of such a system are sometimes not satisfactory 
to the ultimate customer. 

In a second case of image enhancement, the image 



can always be processed. It turns out that an improve- 
ment can usually be made to an image if certain as- 
sumptions are made that are accurate for most cases. 
In an exceptionally large set of images, increasing con- 
5 trast, sharpness, and/or color saturation, will improve 
the image. This model tends to produce better images, 
but the process is unstable, in that for multi-generation 
copying, increases in contrast, saturation, or sharpness 
are undesirable and ultimately lead to a severe image 
10 degradation. Further, the process may undesirably op- 
erate on those images which are good ones. 

Accordingly, we arrive at our third case of image en- 
hancement, a process of automated image enhance- 
ment which operates to vary images which are not per- 
is ceived as good images, but does not operate on images 
which do not need to be improved. 

Many improvements can be made to an image, in- 
cluding luminance enhancement (e.g. US-A 5,450,502); 
sharpness enhancement (e.g., US-A 5,363.209); expo- 
20 sure adjustment (e.g. US-A 5,414,533); color balance 
correction (e.g., US-A 5,357,352, US-A 5,371,615) or 
contrast enhancement (US-A 5,581,370); color satura- 
tion correction (e.g. US-A 5,450,217), etc. These proc- 
esses can be used together in a predictive mode that 
2S does not require iterative processing (e.g. US-A 
5,347,374). Generally, these processing methods oper- 
ate by modifying a set of reproduction curves (TRCs). 
The output image is achieved by using TRC curves, op- 
erating either on the luminance channel of an image ex- 
30 pressed LCjCg coordinates, or preferably on each chan- 
nel in a color density space description of the image in 
Red-Green-Blue (rgb) coordinates. 

Unfortunately, the set of possible color images that 
can be enhanced is large, and varied. Problems are en- 
35 countered in using complex systems having multiple en- 
hancement processors as described, particularly where 
a prior process has reduced continuous tone images to 
an images having only limited number of colors. This 
limited color set images are often encountered in situa- 
te tions where images are retrieved from image archives 
and in situations, where the images are used as back- 
grounds or pictograms. Images with a limited set of 
colors cause problems, because the process algorithms 

assume normal color distributionsrbut when confronted 

45 with limited color set. make inappropriate changes. 
These limited color set images, which are not strictly 
speaking natural color images, do not respond well to 
some of the enhancement operators. On the other hand, 
processor optimized for limited color set images do not 
so operate satisfactorily for natural color continuous tone 
images. Since low color set images provide only unreli- 
able image statistics, image processing operations have 
to be adapted to the statistical uncertainties and not full 
advantage can be taken of the processes when com- 
ss pared to the use of reliable image statistics. It would be 
highly desirable to determine a way to reliably use the 
same processors throughout a more substantial portion 
of the set of color images. 
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In accordance with the present invention, a method 
of controlling an innage enhancement processing com- 
prises creating a signal indicative of the number of 
colors in an image: and using said signal to modify the 
enhancement processing thereof. 

There is provided a method of improving the range 
of color images which can be processed by multiple en- 
hancement processing elements, and improving image 
quality of limited color set images. 

The inventive method determines the number of 
colors that form an image, and with that information, 
adds a control function to one or more of the enhance- 
ment processors, which, based on a number of colors, 
increases and/or decreases the effect of the enhance- 
ment function. In doing so, the range of color images 
which can be handled is increased. 

These and other aspects of the invention wilt be- 
come apparent from the following descriptions used to 
illustrate the preferred embodiment of the invention, 
read in conjunction with the accompanying drawings in 
which: 

Figure 1 shows a block diagram of a system em- 
ploying the present invention; 
Figure 2 shows a block diagram of the processing 
system described; and 

Figures 3 and 4 illustrate exposure control using the 
inventive concept. 

Referring now to the drawings where the showings 
are for the purpose of describing the embodiment of the 
invention and not for limiting same, reference is made 
to Figure 1 and the scan-to-print and archive-to-print 
system shown therein in which the present invention 
may find advantageous use. 

Figure 1 illustrates a scanner 1 0 which may conven- 
iently be a black and white or color scanner which pro- 
duces image signals defined in either RGB space for 
color Images or density space for black and white imag- 
es. Alternatively images may come from an image ar- 
chive 11, where images are not necessarily idealized, 
and may represent unusual scenes or images. These 
images of concern are pictorial in nature, i.e.. they rep- 
resent "natural scenes", but they may be limited in their 
composition or may be problematic with their use. In- 
cluded in this discussion are images used as "back- 
ground " or "wallpaper" in computer displays. Another 
source of images are cliparl archives, which represent 
large sets of inexpensively obtained images, which may 
not be readily available in an output that the user will 
like. While certain computer generated imagery may 
qualify as representing nature scenes, the contemplat- 
ed images are predominantly scanned photographs. 
The images themselves are defined in terms of pixels, 
wherein each pixel has a gray value which varies be- 
tween a white level and a black level. In a currently de- 
sirable system, in which calculations may be done on 8 
bits of information. 256 levels of gray wilt be available 



for use When each separation in r.g.b space is used, 
this allows 24 bits color, and over 16 million colors. Pix- 
els are also identified in terms of position, i.e.. a pixel 
defines a unique area within the image, identified by its 

5 position in a scan line, and the scan line position in a 
page. Color is therefore represented by triplets of gray 
pixels for each color pixel in the image, each triplet of 
gray pixel defining the color in each separation, which 
together form the color pixel. 

10 Color may also be defined by an image file perhaps 
received from another source, using less than 256 levels 
of gray, and in some situations, allowing only 256 colors. 
Additionally, the number of describable colors/gray lev- 
els in an image is only the maximum number of possible 

15 colors and in a large number of cases, only a small sub- 
set of the allowed colors is actually used. This is often 
the case in images that are intended as background im- 
ages, pictograms, etc. or in images that were intended 
for use on the World Wide Web. 

20 The described color images may be directed to an 
automated image enhancement system which will be 
further defined herein. For our purposes, the automated 
image enhancement system may include a segmenta- 
tion system which can identify within a document a type 

25 of image or image object, including pictorial and non- 
pictorial image areas or objects. It will be assumed that 
the output of the automated image enhancement sys- 
tem that is described herein will be directed to a printer, 
CRT, or like device. These devices may have many dis- 

30 tinct characteristics and may be laser printers, or ink jet 
printers or LED displays or CRT displays. However, they 
have as a common requirement the representation of 
gray pictorial images. This may be done with gray print- 
ing or pseudo gray printing. 

35 In terms of deriving data for operation by the present 
image enhancement system, a prescan may be per- 
formed on a document placed on a copying platen and 
scanned by the electro-optical system of the scanner to 
produce a signal representing the document image. Al- 

40 ternatively, the image may be directed to the automated 
image enhancement system from a memory, having 
previously been scanned or derived some other system, 
in which case, the received image is sampled as re- 
quired. The prescan is optionally undersampled, i.e., the 

45 image need not be sampled at the ultimate resolution of 
the system for the purposes of enhancement. In prac- 
tice, it has been determined that a relatively small 
number of pixels representative o1 and dispersed 
through the entire image can accurately represent the 

50 image tor this purpose. In one particular embodiment, 
we use a block of pixels derived from the image^ in ap- 
proximately 512 pixels X 512 pixels. The primary pur- 
pose of this selection is to improve the speed at which 
a software image enhancement system can process the 

55 pictorial images. Sampling at common image resolu- 
tions does not improve the results noted in the inventive 
process herein described significantly, and dramatically 
increases the software processing time required. Some 
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embodiments of the described inventive process might 
decide not to undersample the image, e.g., when the 
Input image size is actually small, or when a hardware 
acceleration is used. 

The input image color signals are edited, or en- 
hanced and prepared for printing at color image 
processing unit 20, which in Figure 1 provides as an out- 
put r, g, b signals. Printer 30 receives the signal from 
color image processing unit 20, and provides for con- 
version of the signals to printer driver signals, commonly 
given in terms of the colorants that will be used in repro- 
duction, typically Cyan, Magenta, Yellow and Key or 
black (CMYK). The printer has an output of the natural 
scene image, which, hopefully, matches user expecta- 
tions of a "good" output image. Alternatively, the proc- 
essed image from color image processing unit 20 is dis- 
played on a television screen. CRT display or the like. 

For the description of the remainder of the process 
of the invention, reference is made to the image of Fig- 
ure 2. Data channel 100 is a bus which carries image 
data, conveniently in terms of rgb color density signals. 
Enhancement channel 1 02 carries data from data chan- 
nel 100 to subsampler 104, which as described, option- 
ally undersamples the data and delivers it to information 
generation and distribution 1 06 which accomplishes the 
common functions required for alt the enhancement 
modules, and which may conveniently be derived at the 
same time. Accordingly, information generation and dis- 
tribution 106 accomplishes conversion of the image 
from RGB space to luminance space - or an approxima- 
tion thereof, derives histograms of signals as required, 
globally and/or locally, and determines such other data 
as required by any enhancement operators used in the 
system. 

The color image data initially received from scanner 
10 or the like, is assumed to be in RGB space initially, 
i.e., red-green-blue space, and for the inventive proc- 
ess, must initially be converted to luminance space (e. 
g. Y in YCiC2 space, a similar luminance/chrominance- 
like value, or an approximation thereof) at information 
generation and distribution 106. It is possible that the 
image will already be in luminance space^ as it is com- 
mon to convert RGB values to luminance/chrominance 
"spacie" f or "oth er image^p rocess in g~ Y C^C^spacens a" 
useful space in which the inventive process can be per- 
formed, and Xerox YES space is one possible embodi- 
ment of such a space. Whatever space is used must 
have a component which relates to the human visual 
perception of lightness or darkness, such as Y of Xerox 
YES of the "Xerox Color Encoding Standard," XNSS 
289005, 1989. In the following, the invention will be de- 
scribed using the Xerox YES color space, although it is 
not limited thereto. 

In addition to the above described operation, at 
block 107, the colors present in the image are counted. 
Generally speaking, continuous tone images will have 
a large number of colors, and limited color set images 
will have less than e.g.: 256 colors. We have selected 



256 as a reasonable threshold for the operations that 
will be described. It will be appreciated that it is possible 
to represent limited color set images in 24 bit color 
space, but the image will not use all ot the possible 
5 colors. On reading the data, a count of the different 
colors is kept, optionally including a color/Black & White 
indicator. A count value, or an indication of whether a 
count exceeds a preselected threshold is output by color 
counter 107. 

10 From information generation and distribution 106, 
data is made available on information channel 108, 
which carries multiple signals to each correction module 
buses 110; 1 20 and 1 30, including the information about 
the number of colors in the image and the optional color/ 

is Black & White indicator. Correction processor buses 
1 1 0, 1 20 and 1 30, respectively, forward signals required 
for operation, tor example, only, to exposure processor 
112, color balance processor 122 and contrast proces- 
sor 132, and likewise return information to information 

20 channel 108 as required. Each of exposure processor 
112, color balance processor 122 and contrast proces- 
sor 132 use the data supplied via information channel 
1 08 to calculate the signals for TRC generators 1 1 4, 1 24 
and 1 34 respectively, to generate TRC's reflecting re- 

25 quired image modification determined at each proces- 
sor. However, in accordance with the invention, the out- 
put of each of the processors is altered based on the 
number of colors found in the image being processed 
as received from 106. Each of TRC generators 114, 124 

30 and 1 34 directs generated TRC's to TRC combiner 1 40, 
which combines the generated TRC's into a single look 
up table and stores the results at TRC controller 1 50 via 
bus 160. TRC controller 150 applies the changes to the 
rgb signals on bus 100 as required. Judge or arbiter 1 70 

55 operates to resolve conflicts in the TRC determination, 
so that processors 112. 1 22 and 1 32 do not operate at 
cross purposes. The output of TRC controller 150 might 
be directed to a sharpness adjustment stage, such as 
that described in US-A 5,363,209. The output of the 

40 TRC controller 1 50 might have a number of bits different 
from the original number of bits in the image. This is spe- 
cially useful if the output is directed to a sharpness ad- 
justment stage that optionally incorporates a reduction 
in the number of bitsrsay from 10 bits to 8 bits: — 

45 It will, of course, be appreciated that the principle of 
the invention applies to any enhancement operation, 
and thus, while only three enhancement processors are 
illustrated here, additional processors or different com- 
binations of process, as described in the background, 

50 may be used. 

Exposure processor 112 and contrast processor 
1 32 both operate on the luminance histogram, and pref- 
erably operate in order of exposure, first, and contrast, 
second. Accordingly, after directing the exposure signal 

55 to TRC generator 114, the exposure signal produced at 
exposure processor 110 is directed to information gen- 
eration and distribution 1 06 via information channel 1 08. 
This signal is used to generate a modified histogram ac- 
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cording to the determined exposure, adjustment. This 
modified histogram is subsequently filtered with a blur 
or smoothing function, such as an averaging over about 
10 adjacent levels, to blur or smooth peaks naturally re- 
sulting in the modified luminance histogram due to ex- 
posure adjustment. It should be noted that the operation 
of this fitter can be modified for increased or decreased 
smoothing. It also appears to be desirable to modify the 
filter operation based on the amount of correction to the 
histogram at the exposure processor. The blurred mod- 
ified histogram is then forwarded to contrast processor 
132, via information channel 108 and correction proc- 
essor bus 1 30. 

In one possible implementation of the invention, a 
map of sensitivity of the respective analyzers/proces- 
sors, (an evaluation that represents the likelihood of im- 
proper processing if the image file is a reduced color set 
image for the used analyzer/processor combinations) 
has determined the following order, from most sensitive 
to least sensitive; 

1 . color balance 

2. sharpness 

3. saturation 

4. contrast 

5. exposure 

For a color implementation, the processor outputs 
are modified with an additional "color map processing 
table" (available in each processor and responsive to 
the color count value) that governs the processing of 
mapped images. Alternatively, the processor might be 
maintained, but modified data, resulting in the desired 
operation, is outputted from the analyzer In this table, 
the following processing is encoded: 

CI. Color Balance: 

For images of 256 colors and less, the color balance 
processing unit is deactivated. 

C2. Sharpness: 

For images of 256 colors and less, the sharpness 
processing is limited to 25% of normal operation., and at 
128 colors all sharpness processing is deactivated. 

C3. Saturation: 

For images of 256 colors or less, saturation 
processing is limited to 50% of normal operation, and at 
128 colors all saturation processing is deactivated. 

C4. Contrast: 

For images of 256 colors or less, contrast is 
switched to "save mode processing". At 128 colors 
processing is deactivated. 



C5. Exposure: 

For images of 64 colors and less, the processing is 
limited to 75% of normal operation, and at 16 colors 
5 processing is deactivated. 

For black & white images, the corresponding 
scheme is used. Here only contrast, exposure and 
sharpness are modified and an example for the settings 
is: 

10 

G1 . Exposure: 

For images with less than 64 levels of gray, expo- 
sure enhancement is reduced to 75% 

15 

G2. Sharpness; 

For images with less than 64 levels of gray, sharp- 
ness enhancement is less than 25%, and at less than 
20 32 levels of gray, sharpness is turned off. 

G3. Contrast: 

For images of 64 levels of gray or less, contrast is 

25 switched to "save mode processing" (see, US A 
5,450,502: processing of images with high global vari- 
ances, specifically incorporated herein by reference), 
and at less than 32 levels of grays, contrast enhance- 
ment is turned off. 

30 In looking at one example of how this process limi- 
tation is obtained, reference is made to US' 5.363,209 
to Eschbach and the changes which would be imple- 
mented therein. Figures 3 and 4 show a flow chart of the 
inventive process at step 300, r, g, b data is received. 

3S At step 302, the r, g, b data is converted to YES values. 
At step 304, histogram signal H(l) is derived from the Y 
signal. At steps 306 and 308 the process branches into 
processing the 100% range and the 90% range of H(l). 
Between steps 308, 310 and 312 the maximum value of 

40 H(l), I lightlOO, andthe minimum valueof H(l), I darklOO, 
is compared respectively to threshold Tlight and T dark 
to determine whether to continue processing the image. 
Depending on the response, (noting the "yes" response 
of step 312 and the "no" response of step 310 that end 

45 the process) a value for gamma is derived by evaluating 
equation 1 2 at steps 322 for the case of the image being 
too light and 320 for the case of the image being too 
dark. At step 324 the preliminary values for gamma 100 
and gamma 90 are compared to find a disagreement, 

so where a disagreement is reached when the difference 
between gamma 90 and gamma 100 is larger than a 
preset value DELTA gamma, where a DELTA gamma 
value of 0.5 has been found to give good results in the 
embodiment outlined in Figures 3 and 4. If a disagree- 

55 ment is detected, the gamma 90 value is ignored and 
only the gamma 100 value is used for further process- 
ing. At step 326, the gamma 100 value is compared to 
a preset value. In this case gamma 100 </= 2 has been 
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mines (107) the number of colors that form an image, 



and with that information, adds a control function to one 
or more enhancement processors (112,122,132), 
which, based on a number of colors, increases and/or 
decreases the effectof the enhancement function. In do- 
ing sp, the range of color images which can be handled 
is increased. 



NATURAL 
SCENE 
IMAGE 



10 



SCANNER 



20 



COLOR 
IMAGE 
PROCESSING 



g, b 



r 



30 



PRINTER 
(CMYK) 



NATURAL 
SCENE 
REPRO- 
DUCTION 



CO 
< 

00 

^. 

00 

o 

Q. 
liJ 



FIG. 1 



Printed by Jouve. 75001 PARIS (FR) 



BNSDOCID: <EP 0878777A3_I_> 



EP 0 878 777 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Appticatton Nuntoer 

EP 98 30 3591 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Citation o» document with i no i cation, wnere apof cpnate, 
of f*3levant passages 



Reievan: 
to claim 



CtASSiFICATlON OF THE 
APPLICATION ilnt.CI.6) 



IeP 0 388 877 A (KONISHIROKU PHOTO IND) 
i 26 September 1990 
I * abstract * 

I* page 2. line 40 - page 3, line 16 * 
i* page 4, line 14 - line 37 * 

EP 0 413 557 A (SONY CORP) 
20 February 1991 
* abstract * 

*■ column 2, line 5 - line 27 * 



* page 5, line 26 - line 42; figure 4 * 



; PATENT ABSTRACTS OF" JAPAN 
vol. 095, no. 008, 29 September 1995 
& JP 07 131807 A (NIPPON HOSO KYOKAI 
;OTH£RS: 01). 19 May 1995 
^ abstract * 



I he present search report has been drawn up tor all claims 



1-3 



! G06T5/40 



1-3 



1-3 



TECHNrCAL FIELDS 
SEARCHED tlnt.Cl.6i 



G06T 



Place ot ;«arcrt 

THE HAGUE 



Date cr complennn :»( me searcn 

29 October 1998 



Gonzalez Ordonez, 0 



CATEGOPV OF CITED DOCUMENTS 



pania.iiar*y r-?(#vant ff taKen alone 
y panfCuiarf/ retevant it ccmoinea wan ancther 

ooCLdTiwut ol the sarrw category 
A techoologtcai backgrourd 
O : non-wntten cisctoaure 
P int»rni©oiat9 document 



T • th*»cry or princes!© underlying the invention 
E ; earlier patent ^ocum&nt. du Duciisnea on, or 

atter :ne filing date 
O dccumertf cited *n the application 
L . document cited (or other reasons 

A ; member cf the %An& patent (amify, correspsrtfjng 



BNSDOCID- <EP 0878777A3_I_> 



2 



